In video image processing it is a common requirement to improve the sharpness of an image, or to enhance the definition of edges. Many video processing techniques can cause the loss of high frequency information from the images causing a softening or blurring effect. The use of a suitable enhancement process can replace the missing information and increase the apparent quality of an image.
There are numerous enhancement techniques in use, with a variety of names. Terms such as “edge enhancement” and “sharpening” describe processes that, while they may vary in details, perform essentially similar functions.
U.S. Pat. No. 4,758,891 (Hitchcock et al., Philips Corp.) and U.S. Pat. No. 4,994,915 (Takahashi et al., Hitachi Ltd.) both demonstrate the use of delays to create a “correction signal” which is combined with the original image to produce an enhanced output. Circuits similar to these where the second derivative of the input signal, or an approximation to it, is used to generate the correction signal, form the basis of many image enhancement systems currently in use. The use of a 2nd derivative correction signal is illustrated in FIGS. 1A-1D.
FIG. 1A shows a signal representing a step change in image brightness. This is typical of a section through an edge in the image. In FIG. 1B, previous processing has damaged the signal so that the transition is no longer abrupt, but changes gradually from one brightness level to another. The gradual change in brightness causes the edge to appear blurred rather than sharp. FIG. 1C shows a correction signal generated from the second derivative of the soft edge. FIG. 1D shows the soft edge, after the correction signal has been subtracted. It can be seen that the signal transition from dark to light has been accelerated.
FIGS. 2A-2D show the effect of a gain control that adjusts the amount of correction that is used. It can be seen that increasing the amount of correction increases the speed of transition (seen as a steeper gradient close to the centre of the edge), with FIG. 2C showing the onset of overshoot caused by excessive gain.
Overshoot caused by over-enhancement is generally something to be avoided, as it can give rise to unpleasant ‘halo’ effects around edges. However, the increased transition speed that comes with such levels of enhancement is desirable.
Clamping is commonly used to restrict the swing of the enhanced signal, permitting higher gains to be used while eliminating overshoot. FIG. 2D shows the effect of clamping, where the signal of FIG. 2C has been clamped to the minimum and maximum values of the input signal. One possible implementation of clamping is to measure the minimum and maximum values of the original signal in the local area, and limit the enhanced signal to that range. This is illustrated in FIG. 4, where the shaded area represents the clamping limits measured using a five pixel wide moving window.
U.S. Pat. No. 6,363,526 (Vlahos, Berman, Dadourian, Ultimatte Corp.) and U.S. Pat. No. 6,278,494 (Inamura, Kanai, Masuda, Canon KK.) describe systems that limit the range of the enhanced signal so as to avoid overshoot.
It is known that a small, carefully controlled amount of overshoot can add significantly to the apparent sharpness of an image, but is not perceived as a visible halo. U.S. Pat. No. 6,363,526 discloses the idea of adding an offset to the measured maximum value of the input signal, and subtracting an offset from the measured minimum, in order to preserve small spikes in the enhanced output.
A further reason to avoid overshoot is that video standards give guidelines on the maximum and minimum values of the video signal. For example, in a YCbCr system such as ITU-R BT601, the luminance channel (Y) may be represented using 8 bit values between 0 and 255. A value of 16 represents black, and a value of 235 represents white. Signal excursions below 16 and above 235, while not strictly prohibited, are discouraged, and values below zero or greater than 255 are, obviously, not representable.
From the discussion of the prior art, it is clear why overshoot control is required. However, a system that eliminates overshoot by clamping to a local minimum and maximum may not provide the flexibility of control that is required.
A sharpening algorithm can have undesirable side effects, such as increasing noise, or adding “graininess” to the image, and those side effects tend to increase with gain. Overshoot can increase the perception of sharpness in an image. Allowing a small amount of overshoot may therefore allow gain to be reduced while retaining the same amount of apparent sharpness, or to allow the apparent sharpness to be increased with no additional gain.
Where the goal is “edge enhancement”, rather than “sharpening”, variations on the methods described in the prior art may be used to add strong correction to particular types of image feature, e.g., high contrast edges. In these situations some viewers may even prefer the visual effect of halos caused by a large amount of overshoot.
It is therefore apparent that a flexible overshoot control mechanism should allow overshoot to be eliminated altogether, or for controlled quantities of overshoot, ranging from a little to a lot, to be used depending upon the application.
In a system that allows overshoot, care must be taken to ensure that it does not adversely affect image quality in areas where it is not required. For example, a large amount of overshoot may be desirable on large edge transitions, but the same amount of overshoot may be inappropriate in areas of small detail, where it may exaggerate texture or noise. In other cases, an overshoot applied in an area of highlight or deep shadow may cause the signal to saturate, losing detail in the image.
While permitting overshoot, it is also necessary to observe signal excursion restrictions, such as the 16-235 value range in ITU-R BT601 YCbCr encoding.
Embodiments of the present invention permit an amount of overshoot proportional to the size of the feature being enhanced (ie the step in amplitude at the edge). Thus, the amount of overshoot is reduced towards zero as the feature size is reduced to avoid the emphasis of noise. In preferred embodiments the constant of proportionality, which is applied to the size of the feature to determine the permitted amount of overshoot, can be adjusted. Such embodiments enable the amount of overshoot to be controlled to be appropriate for the feature being enhanced.
Further embodiments of the present invention compress the permitted amount of overshoot depending on the limit of the channel (i.e. permitted signal range) and the position of the original signal with respect to those limits. For example, for mid-range signals, the normal amount of overshoot is permitted and for signals closer to the limits of the channel, e.g. almost black or almost white, the amount of overshoot is reduced towards zero. In further embodiments, the compression is used to limit the signal excursion according to the encoding standard (e.g. within the range 16-235). Such embodiments prevent the saturation of the signal.
The invention is defined in its various aspects with more precision in the appended claims to which reference should now be made.